MPI Performance Comparison on Distributed and Shared Memory Machines

The widely implemented MPI Standard 10] deenes primitives for point-to-point inter-processor communication (IPC), collective IPC, and synchronization based on message passing. The main reason to use a message passing standard is to ease the development, porting, and execution of applications on the variety of parallel computers that can support the paradigm, including shared memory, distributed memory, and shared memory array multiprocessors. This paper compares the SGI Power Challenge, a shared memory multiprocessor, with the Intel Paragon, a distributed memory machine. This paper addresses two questions: How do MPI and memory performance compare on the SGI Power Challenge for various message sizes and numbers of CPUs? Can MPI's relative performance on the SGI Power Challenge also be used to predict performance on shared memory, distributed memory, and shared memory array multiprocessors? Memory and communications tests written in C++ using messages of double precision arrays show that both memory and MPI blocking IPC performance on the Power Challenge degrade once total message sizes grow larger than the second level cache. Comparing the MPI and memory performance curves indicate Power Challenge native MPI point-to-point communication is implemented using memory copying and synchronization is implemented using a binary tree algorithm. A model of blocking IPC for the SGI Power Challenge was developed, which is validated with performance results. A new measure of communications eeciency and overhead, the ratio of IPC time to memory copy time, is used to compare relative IPC performance for diierent machines. Comparison of Power Challenge and the Paragon show that the Paragon is more eecient for small messages, but the Power Challenge is better for large messages. Power Challenge observations do not generally correspond with those on the Paragon, indicating shared memory multiprocessor results should not be used to predict distributed memory multiprocessor performance. This suggests that parallel algorithms should not judged solely on their performance on one type of machine.